Extracting Triangular 3D Models, Materials, and Lighting From Images

Is it really possible to frame this process as an inverse rendering task, and optimize as many steps as possible jointly, driven by the quality of the rendered images of the reconstructed model, compared to the captured input imagery?

Table of Contents

Busting the Jargon

Photogrammetry is the art, science, and technology of obtaining reliable information about physical objects and the environment through recording, measuring, and interpreting photographic images and patterns of recorded radiant electromagnetic energy and other phenomena. For more information regarding the usage of this technique to capture 3D models, you can refer to the articles From images to 3D models and Photo tourism: exploring photo collections in 3D.
Inverse Rendering refers to the estimation of intrinsic scene characteristics given a single photo or a set of photos of the scene. While predicting these characteristics from 2D projects is highly underconstrained, recent advances have made a big step in solving this problem. For a quick primer into inverse rendering, you can check out this amazing video from ICCV 2015.
Deformable Tetrahedral Meshes or DefTet is a particular parameterization of 3D shapes that utilizes volumetric tetrahedral meshes for the reconstruction problem. It optimizes for both vertex placement and occupancy and is differentiable with respect to standard 3D reconstruction loss functions. It is thus simultaneously high-precision, volumetric and amenable to learning-based neural architectures. For more information, refer to the paper Learning Deformable Tetrahedral Meshes for 3D Reconstruction.
High Dynamic Range Rendering, or HDRR, is the real-time rendering and display of virtual environments using a dynamic range of $65535:1$ or higher (used in gaming and entertainment technology). This allows the preservation of details lost due to limiting contrast ratios. For more information on HDRR, you can check out the corresponding Wikipedia article.
A signed Distance Field, or SDF, is a function that takes a position as an input and outputs the distance from that position to the nearest part of a shape. For more information on SDF, you can refer to this article.
BRDF, or Bidirectional Reflectance Distribution Function, is a function of four real variables that defines how light is reflected at an opaque surface. It is employed in the optics of real-world light, in computer graphics algorithms, and in computer vision algorithms. For more information on BRDF, you can check the corresponding Wikipedia article.

Existing Approaches and Their Limitations

Multi-view 3D Reconstruction

Classical Methods

Classical methods for multi-view 3D reconstructions, such as Building Rome in a day and Pixelwise View Selection for Unstructured Multi-View Stereo, exploit inter-image correspondences to estimate depth maps. These methods typically fuse depth maps into point clouds, optionally generating meshes. They rely heavily on the quality of matching, and errors are hard to rectify during post-processing.
Another set of classical methods, such as Poxels: Probabilistic Voxelized Volume Reconstruction, uses voxel grids to represent shapes. These methods estimate occupancy and color for each voxel and are often limited by the cubic memory requirement.

Neural Implicit Representations

Neural Radiance Fields and its follow-up works use volumetric representations and compute radiance by ray marching through a neurally encoded 5D light field. While achieving impressive results on novel view synthesis, geometric quality suffers from the ambiguity of volume rendering.
Surface-based rendering methods such as Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision, and Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance use implicit differentiation to obtain gradients, optimizing the underlying surface directly. Unisurf is a hybrid approach that gradually reduces the sampling region, encouraging a volumetric representation to converge to a surface. NeuS provides an unbiased conversion from a signed distance field into density for volume rendering. The problem with all these methods is their reliance on ray marching for rendering, which is computationally expensive during training and inference.
Volumetric and implicit shape representations such as SDFs and NeRFs can be converted to meshes through Marching Cubes in a post-processing step. However, Marching Cubes inevitably imposes discretization errors. As a result, the output mesh quality, particularly at the moderate triangle counts typically used in real-time rendering, is often not sufficient.

Explicit Surface Representations

Lighting Estimation

Older works on Bidirectional Texture Function (BTF) and Spatially varying BRDF (SVBRDF) estimation rely on special viewing configurations, lighting patterns, or complex capturing setups.
More recent methods, such as MaterialGAN, use neural networks to predict BRDF from images. Differentiable rendering methods such as DIB-R++ and Appearance-Driven Automatic 3D Model Simplification learn to predict geometry, SVBRDF, and, in some cases, lighting via 2D image loss. Still, their shape is generally deformed from a sphere and cannot represent arbitrary topology.
Neural implicit representations successfully estimate lighting and BRDF from image collections. Neural Reflectance Fields for Appearance Acquisition and NeRV model light transport support advanced lighting effects, e.g., shadows, but have high computational costs.

The Approach in Details

Overview

The authors present a method for 3D reconstruction supervised by multi-view images of an object illuminated under one unknown environment lighting condition, together with known camera poses and background segmentation masks.
The target representation consists of triangle meshes, spatially-varying materials (stored in 2D textures), and lighting (a high dynamic range environment probe).
We carefully design the optimization task to render triangle meshes while robustly handling arbitrary topology explicitly.
The authors adapt DMTet to work in the setting of 2D supervision and jointly optimize shape, materials, and lighting.
At each optimization step, the shape representation, i. e, parameters of a signed distance field (SDF) defined on a grid with corresponding per-vertex offsets – is converted to a triangular surface mesh using a marching tetrahedra layer.
Next, the extracted surface mesh is rendered in a differentiable rasterizer with deferred shading and computed loss in image space on the rendered image compared to a reference image.
Finally, the loss gradients are back-propagated to update the shape, textures, and lighting parameters.

An overview of the entire inverse rendering pipeline proposed by the authors of Extracting Triangular 3D Models, Materials, and Lighting From Images

Problem Formulation

$\phi$ denotes our optimization parameters (i.e., SDF values and vertex offsets representing the shape, spatially varying material, and light probe parameters).
$c$ is a given camera pose.
$I_{\phi}(c)$ is the image produced by the differentiable renderer for the given camera pose $c$.
$I_{\mathrm{ref}}(c)$ is the reference image which corresponds to the view from the same camera.
$L$ is the loss function given by $L = L_{image} + L_{mask} + \lambda L_{reg}$ where,
$L_{image}$ is an image space loss which is basically the $L_{1}$ norm of the tone mapped colors
$L_{mask}$ is a mask loss which is basically a squared $L_{2}$ loss, and
$L_{reg}$ is a regularization loss for regularizing the SDF values of DMTet to reduce floaters and internal geometry

Learning the Topology

The proposed rendering pipeline explicitly renders triangle meshes during optimization in order to avoid discretization errors posed by Marching Cubes in a post-processing step.
The authors leverage Deep Marching Tetrahedra or DMTet in a 2D supervision setting through differentiable rendering. DMTet is a hybrid 3D representation that represents a shape with a discrete SDF defined on vertices of a deformable tetrahedral grid.
The SDF is converted to triangular mesh using a differentiable marching tetrahedra layer.
The loss is computed on renderings of the 3D model and is back-propagated to the implicit field to update the surface topology. This allows us to optimize the surface mesh and rendered appearance end-to-end directly.
Given a tetrahedral grid with vertex positions $v$, DMTet learns SDF values $s$, and deformation vectors $\Delta v$.
The SDF values and deformations can either be stored explicitly as values per grid vertex or implicitly by a neural network.
At each optimization step, the SDF is first converted to a triangular surface mesh using Marching Tetrahedra, which is shown to be differentiable w.r.t. SDF and can change surface topology in DMTet.
Next, the extracted mesh is rendered using a differentiable rasterizer to produce an output image. Image-space loss gradients are back-propagated to the SDF values and offsets (or network weights).
A neural SDF representation can act as a smoothness prior, which can be beneficial in producing well-formed shapes. Directly optimizing per-vertex attributes, on the other hand, can capture higher frequency detail and is faster to train. In practice, the optimal choice of parametrization depends on the ambiguity of geometry in multi-view images.

A representation of Marching Tetrahedra extracting faces from a tetrahedral grid.

Shading Model

The authors follow Appearance-Driven Automatic 3D Model Simplification for differentiable rendering and the PBR material model from Practical Physically Based Shading in Film and Game Production in order to easily be able to import game assets and render the optimized models directly in existing engines without modifications.
The proposed rendering pipeline optimizes mesh topology, which requires continually updating the parametrization, potentially introducing discontinuities into the training process. To robustly handle texturing during topology optimization, the authors use volumetric texturing and use world space position to index into our texture. This ensures that the mapping varies smoothly with vertex translations and changing topology.
In order to manage the high memory footprint of volumetric textures that grows cubically, the authors extend the approach proposed by PhySG, using a multilayer perceptron (MLP) to encode all material parameters in a compact representation. This representation can adaptively allocate detail near the 2D manifold representing the surface mesh, a small subset of the dense 3D volume. Formally speaking, the authors let a positional encoding + MLP represent a mapping $\mathbf{X} \rightarrow\left(\boldsymbol{k}_{d}, \boldsymbol{k}_{\mathrm{Orm}}, \mathbf{n}\right)$, i.e, given a world space position $\mathbf{x}$, compute the base color, $\boldsymbol{k}_{d}$, specular parameters, $\boldsymbol{k}_{\mathrm{Orm}}$ (roughness, metalness), and a tangent space normal perturbation given by $\mathbf{n}$.
Once the topology and MLP texture representation have converged, the model is re-parametrized. The authors generate unique texture coordinates using xatlas and sample the MLP on the surface mesh to initialize 2D textures, then continue the optimization with fixed topology.

Image-Based Lighting

The first term represents the integral of the specular BSDF with a solid white environment light. It only depends on the parameters $cos \theta = \omega_{i}n$ and the roughness $r$ of the BSDF and can be precomputed and stored in a 2D lookup texture.
The second term represents the integral of the incoming radiance with the specular NDF $D$. This term is also pre-integrated and represented by a filtered cubemap. In each mip-level, the environment map is integrated against D for a fixed roughness value.

Experiments and Results

Training and Validation

3D Model Extraction

Scene Editing and Simulation

Relighting Quality of Reconstructed Model

Deployment of Learned Representation

Reconstruction from Photographs

Comparison between Spherical Gaussians and Split Sum Approximations

Results on the NeRF Synthetic Dataset

Limitations of the Approach

The authors find that the main limitation of the proposed differentiable inverse rendering pipeline is the simplified shading model that doesn't account for global illumination or shadows. This choice is intentional to accelerate optimization, but is a limiting factor for material extraction and relighting. However, with the current progress in differentiable path tracing, the authors hope that this limitation would be lifted in future work.
The proposed approach relies on alpha masks to separate foreground from background. While the method seems quite robust to corrupted masks, it would be beneficial to further incorporate this step into the system.
The proposed renderer uses a differentiable rasterizer with deferred shading, hence reflections, refractions, and translucency are not supported.
During optimization, the proposed renderer only renders direct lighting without shadows.

Conclusion

In this report, we discussed the paper Extracting Triangular 3D Models, Materials, and Lighting From Images which proposes a highly efficient inverse rendering pipeline capable of extracting triangular meshes of unknown topology with spatially-varying materials and lighting from multi-view images.
We discuss existing approaches to solve this problem, including classical methods such as Building Rome in a day, explicit surface representations such as DMTet, and neural implicit representation approaches such as Neural Radiance Fields.
We discussed an overview of the differentiable inverse rendering pipeline, including the problem formulation, topology learning, shading model, and image lighting.
We demonstrated the performance of the proposed pipeline for 3D model extraction, scene manipulation, simulation scenarios, and reconstruction from photographs. We have seen that results are on par with state-of-the-art view synthesis and material factorization while directly optimizing an explicit representation: triangle meshes with materials and environment lighting.
The representation learned by the proposed pipeline is, by design, directly compatible with modern 3D engines and modeling tools, which enables a vast array of applications and simplifies artist workflows.
The authors note the limitations of the proposed approach and discuss the possibilities of future works addressing these issues.
The authors note that apart from deep fakes, common to all scene reconstruction methods, there's little possibility of nefarious use cases of the proposed method.

Similar Reports